All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

摘要

Deep neural network is difficult to train and this predicament becomes worseas the depth increases. The essence of this problem exists in the magnitude ofbackpropagated errors that will result in gradient vanishing or explodingphenomenon. We show that a variant of regularizer which utilizes orthonormalityamong different filter banks can alleviate this problem. Moreover, we design abackward error modulation mechanism based on the quasi-isometry assumptionbetween two consecutive parametric layers. Equipped with these two ingredients,we propose several novel optimization solutions that can be utilized fortraining a specific-structured (repetitively triple modules of Conv-BNReLU)extremely deep convolutional neural network (CNN) WITHOUT any shortcuts/identity mappings from scratch. Experiments show that our proposed solutionscan achieve distinct improvements for a 44-layer and a 110-layer plain networkson both the CIFAR-10 and ImageNet datasets. Moreover, we can successfully trainplain CNNs to match the performance of the residual counterparts. Besides, we propose new principles for designing network structure from theinsights evoked by orthonormality. Combined with residual structure, we achievecomparative performance on the ImageNet dataset.